Maintaining hardware resource bandwith quality-of-service via hardware counter

ABSTRACT

Each time a currently scheduled virtual machine (VM) accesses a hardware resource over a bus for the hardware resource via the currently scheduled VM running on a processor, a hardware component adjusts a bandwidth counter associated with usage of the bus for the hardware resource, without involvement of the currently scheduled VM or a hypervisor managing the currently scheduled VM. Responsive to the bandwidth counter reaching a threshold value, the hardware component issues an interrupt for handling by the hypervisor to maintain bandwidth quality-of-service (QoS) of bus bandwidth related to the hardware resource. Upon expiration of a regular time interval prior to the bandwidth counter reaching the threshold value, the hardware component resets the bandwidth counter to a predetermined value associated with the currently scheduled VM, without involvement of the currently scheduled VM or the hypervisor; the hardware component does not issue an interrupt. The hardware resource can be memory.

BACKGROUND

A computing device typically runs one instance of an operating systemthat has access to the hardware resources of the computing device.However, a technique known as virtualization permits a computing deviceto run a number of instances of operating systems. In virtualization,the computing device instantiates a number of such virtual machines(VMs), and each VM runs its own instance of an operating system.

SUMMARY

A method of an embodiment of the disclosure includes, each time acurrently scheduled virtual machine (VM) accesses a hardware resourceover a bus for the hardware resource via the currently scheduled VMrunning on a processor, a hardware component adjusting a bandwidthcounter associated with usage of the bus for the hardware resource,without involvement of the currently scheduled VM or a hypervisormanaging the currently scheduled VM. The method includes, responsive tothe bandwidth counter reaching a threshold value, the hardware componentissuing an interrupt by for handling by the hypervisor to maintainbandwidth quality-of-service (QoS) of bus bandwidth related to thehardware resource. The hardware resource and the hardware component areeach hardware different than the processor. The bandwidth counter isimplemented directly in hardware as opposed to being asoftware-implemented counter.

A method of another embodiment of the disclosure includes, each time acurrently scheduled VM accesses a hardware memory over a bus for thehardware memory via the currently scheduled VM running on a processor, ahardware memory controller decrementing a bandwidth counter associatedwith usage of the bus for the hardware resource, without involvement ofthe currently scheduled VM or a hypervisor managing the currentlyscheduled VM. The method includes, responsive to the bandwidth counterreaching zero, the hardware memory controller issuing an interrupt forhandling by the hypervisor to maintain bandwidth QoS of bus bandwidthrelated to the hardware memory. The bandwidth counter is implementeddirectly in hardware as opposed to being a software-implemented counter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The drawings referenced herein form a part of the specification.Features shown in the drawing illustrate only some embodiments of thedisclosure, and not of all embodiments of the disclosure, unless thedetailed description explicitly indicates otherwise, and readers of thespecification should not make implications to the contrary.

FIG. 1 is a diagram of an example computing system in which hardwareresource bandwidth quality-of-service (QoS) is maintained via a hardwarecounter.

FIG. 2 is a flowchart of an example method for maintaining hardwareresource bandwidth QoS via a hardware counter in the computing system ofFIG. 1.

FIG. 3 is a diagram of another example computing system in whichhardware resource bandwidth QoS is maintained via a hardware counter,and which is consistent with but more particular than the computingsystem of FIG. 1.

FIG. 4 is a flowchart of an example method for maintaining hardwareresource bandwidth QoS via a hardware counter in the computing system ofFIG. 3, and which is consistent with but more particular than the methodof FIG. 2.

DETAILED DESCRIPTION

The following detailed description of exemplary embodiments of thedisclosure refers to the accompanying drawings that form a part of thedescription. The drawings illustrate specific exemplary embodiments inwhich the disclosure may be practiced. The detailed description,including the drawings, describes these embodiments in sufficient detailto enable those skilled in the art to practice the disclosure. Thoseskilled in the art may further utilize other embodiments of thedisclosure, and make logical, mechanical, and other changes withoutdeparting from the spirit or scope of the disclosure. Readers of thefollowing detailed description should, therefore, not interpret thedescription in a limiting sense, and only the appended claims define thescope of the embodiment of the disclosure.

As noted in the background section, virtualization is a technique thatpermits a computing device to run more than one instance of an operatingsystem on the same computing device as if there were multiple computingdevices, which are referred to as virtual machines (VMs). The VMstypically share hardware resources, like memory and processors, of agiven computing device. A VM manager (VMM), or hypervisor, manages theVMs, ensuring that they appropriately share the hardware resources ofthe computing device in accordance with certain reservations,allocations, or priorities, which are referred to generally under theumbrella term or phrase quality-of-service (QoS), assigned to the VMs.

While for a given hardware resource like hardware memory, the totalamount of the memory of the computing device that is allocated to agiven VM can be reserved, a related metric, bus bandwidth for or relatedto such memory, cannot as easily be specified. When a VM runs on aprocessor of a computing device, the VM accesses hardware memory over abus that may interconnect a memory controller, such as a memorymanagement unit (MMU) to both the hardware memory and the processor. Assuch, accesses of the memory by the VM, by virtue of it running on theprocessor in question, uses bandwidth of the bus. Ensuring that each VMis given the appropriate amount of bandwidth, however, is difficult toachieve, particularly without involving the VMs themselves in acooperative manner, and/or without involving the hypervisor itself.Employing such software-oriented solutions to manage memory bandwidthQoS can result in undesirable performance degradation of the VMs.

Techniques disclosed herein, by comparison, provide for ahardware-centric—and indeed, in at least some implementations, ahardware-only—solution to managing to bus bandwidth related to hardwarememory and other hardware resources to maintain such bus bandwidth QoS.A hardware register directly accessible by a particular hardwarecomponent like a memory controller, without involving software like ahypervisor or a VM, can be set to a predetermined value associated withthe VM that is currently scheduled for running on a processor. When theVM begins, or just prior to beginning, running on the processor, thepredetermined value is copied to a bandwidth counter from the hardwareregister. The bandwidth counter itself may be a different hardwareregister that is directly accessible by the hardware component inquestion.

As the VM accesses the hardware memory or other hardware resource inquestion over a corresponding bus, the hardware component adjusts abandwidth counter associated with usage of the bus. Such adjustment canbe a decrementing, for instance, and does not involve either thecurrently scheduled VM, the hypervisor managing this VM, or any otherVM. The VMs indeed may be unaware and have no knowledge that their busbandwidth for the hardware resource is being tracked.

If prior to a regular time interval, such as one millisecond, elapsingthe bandwidth counter reaches a threshold value, the hardware componentissues an interrupt. The threshold value may be zero in the case wherethe bandwidth counter is decremented. The hypervisor handles thisinterrupt, which is typically a hardware interrupt, so that QoS of busbandwidth related to the hardware resource in question is maintained.For instance, the hypervisor may switch the currently scheduled VM for adifferent VM to run on the processor, even before the regular timeinterval has elapsed, because the currently scheduled VM has used up itsbus bandwidth allocation (corresponding to the predetermined value forthe VM that was copied from the hardware register to the bandwidthcounter) for the regular time interval.

By comparison, if the regular time interval elapses without thebandwidth counter reaching the threshold value, the hardware componentresets the bandwidth counter, by copying the predetermined value for theVM from the hardware register to the bandwidth counter. The hypervisoris not involved in the resetting of the bandwidth counter in this case,and the hardware component does not issue any interrupt. As such, thebus usage for the hardware resource in question is tracked completely inhardware, via a bandwidth counter that is implemented directly inhardware (as opposed to being a software-implemented counter).Therefore, QoS maintenance does not involve the hypervisor unless anduntil the current VM uses its permitted bandwidth, resulting in thehardware component issuing an interrupt.

This process is repeated, until an interrupt is generated, or until itis time for the current VM to be switched for a new VM to run on theprocessor as part of the regular schedule for the processor. In thelatter case, the hypervisor copies a predetermined value correspondingto the new VM's bus bandwidth allocation to the hardware register, andthe process described above is started again for the new VM to run onthe processor, by copying the predetermined value from the hardwareregister to the bandwidth counter, adjusting the counter each time thehardware resource is accessed over the bus, and so on. Each VM can havea different predetermined value corresponding to its bus bandwidthallocation for the hardware resource.

The hypervisor is thus involved just—and in at least someimplementations, only—at two times: when the hardware component issuesan interrupt, and when the VM currently running on the processor is tobe switched for a different processor. Therefore, the techniquesdescribed herein do not require much if any overhead on the part of thehypervisor, since the hypervisor does not itself keep track of bandwidthusage of the hardware resource by the VMs while such bandwidth is beingused; rather, the hardware component does. Furthermore, as noted above,the VMs do not have to be and indeed in at least some implementationsare not aware that their bandwidth usage is being tracked.

FIG. 1 shows an example computing system 100, which may be implementedon a single computing device, such as a single computer, or overmultiple computing devices, such as multiple computers. The computingsystem 100 includes hardware 102 and software 104. The software 104 isimplemented by the hardware 102, as indicated by brace 105, insofar asthe hardware 102 executes the software 104.

The hardware 102 of the computing system 100 includes at least aprocessor 106, a hardware component 108, and a hardware resource 110.The processor 106, the hardware component 108, and the hardware resource110 are physically connected to one another over a hardware bus 112 thatis related to and/or is for the hardware resource 110. For example, thehardware component 108 may be an MMU or other memory controller when thehardware resource 110 is hardware memory, such that the bus 112 is ahardware memory bus. The hardware 102 can and typically does includeother hardware, in addition to the processor 106, the hardware component108, the hardware resource 110, and the bus 112 that are depicted inFIG. 1. As just one example, there can be processors in addition to theprocessor 106.

In the in the example of FIG. 1, the processor 106 accesses the hardwareresource 110 over the hardware bus 112 directly. Where the hardwareresource 110 is hardware memory, such an implementation may be referredto as a direct memory access (DMA) approach by which the processor 106accesses the hardware memory. In other implementations, however, accessof the hardware resource 110 by the processor 106 may occur through thehardware component 108 itself. It is noted that the terminologyprocessor 106 as used herein can encompass a single core of amultiple-core processor, where this core is a virtual or actual physicalprocessor core. For instance, some processors have two or more physicalcores, and each physical core can be divided into two or more virtualcores. The terminology processor thus encompasses each such physicalcore and/or each such virtual core of a multiple-core processor.

The hardware 102 of the computing system 100 also includes a bandwidthcounter 114 that the hardware component 108 adjusts to track usage ofthe bus 112 when the processor 106 accesses the hardware resource 110.For instance, the hardware component 108 may decrement the bandwidthcounter 114 each time the processor 106 accesses the hardware resource110. The bandwidth counter 114 may be a hardware register, and isdirectly accessible by the hardware component 108 without the assistanceor involvement of any of the software 104 of the computing system 100.The bandwidth counter 114 may be a part of the hardware component 108itself, or of the processor 106 or of the hardware resource 110 itself,such as a hardware register thereof, although the counter 114 is calledout separately in FIG. 1 for illustrative clarity.

The software 104 of the computing system 100 includes a hypervisor 116and a number of VMs 118, such as at least one VM 118. The hypervisor 116manages instantiation and deletion of the VMs 118, and further managesscheduling of the VMs 118 for running on the processor 106. Thehypervisor 116 is also referred to as a VMM. Each virtual machine 112 isa software implementation of an actual physical machine—i.e., a computeror computing device—that executes software as if it were a separatephysical machine. Each VM 118 typically includes its own operatingsystem, which is referred to as a guest operating system, and whichgoverns just the VM 118, and not other VMs 118 that may be running onthe same host computing system 100 in question.

FIG. 2 shows an example method 200 by which bus bandwidth of thehardware resource 110 is managed for QoS purposes in a hardware-centricmanner within the computing system 100. The method 200 can operatewithin a scheduling methodology in which the hypervisor 116 schedulesdifferent VMs 118 to run on the processor 106 at different times. Inparticular, there can be regular time intervals, such as one millisecondeach. The hypervisor 116 may schedule a first VM 118 to run on theprocessor 106 for X time intervals, followed by a second VM 118 to runon the processor 106 for Y time intervals, and so on, where X may beequal to or different than Y. However, if the first VM 118 uses itsallocated bus bandwidth for the hardware resource 110 within a giventime interval, the hypervisor 116 may switch out the first VM 118 forthe second VM 118 to run on the processor 106 earlier, before theprocessor 106 has used its X time intervals of processor time.

The hypervisor 116 thus schedules a VM 118 to run on the processor 106(202). This VM 118 is referred to as the currently scheduled VM 118. Apredetermined value for the currently scheduled VM 118 is copied to thebandwidth counter 114 (204). The predetermined value is associated withthe currently scheduled VM 118, and corresponds to the amount ofbandwidth of the bus 112 the VM 118 is permitted to use during eachregular time interval while the VM 118 runs on the processor 106 whenaccessing the hardware resource 110. That is, the predetermined valuecorresponds to the bus bandwidth QoS of the currently scheduled VM 118for the hardware resource 110, during each regular time interval. Aregular time interval thus starts (206).

Each time the currently scheduled VM 118, as running on the processor106, accesses the hardware resource 110 over the bus 112, the hardwarecomponent 108 correspondingly adjusts the bandwidth counter 114 (208).If the bandwidth counter 114 reaches a threshold value as a result ofsuch adjustment (210), then the hardware component 108 issues aninterrupt, such as a hardware interrupt (212). The interrupt signals tothe hypervisor 116 that the currently scheduled VM 118 is to be switchedout on the processor 106 for a different, new VM 118 (i.e., a newlyscheduled VM 118), such that the method 200 is repeated at part 202.

If, however, the bandwidth counter 114 has not reached the thresholdvalue (210), and if the current regular time interval has not yetelapsed or expired (214), then the method 200 proceeds back to part 208.If the bandwidth counter 114 does not reach the threshold value duringthe current regular time interval, then the method 200 proceeds frompart 210 to part 214, and then to part 216. If it is not yet time toschedule a new VM 118 (i.e., a newly scheduled VM 118) for running onthe processor 106, then the hardware component 108 resets the bandwidthcounter 114 to the predetermined value for the currently scheduled VM118 (218), and the method 200 proceeds back to part 206 with a newregular time interval starting.

As such, if the currently scheduled VM 118 does not ever exceed its QoSfor bus bandwidth related to the hardware resource 110—such that thebandwidth counter 114 never reaches the threshold value during any timeinterval when this VM 118 is running on the processor 106—at some pointit will be time to schedule a new VM 118 on the processor 106 forrunning on the processor 106. At this point in time, the method 200proceeds from part 216 to part 212, at which the hardware component 108may issue a hardware interrupt also signaling to the hypervisor 116 thatit is time to switch out the VM 118 that is running on the processor106. For instance, in the example noted above where a first VM 118 isscheduled to run on the processor 106 for X regular time intervals, onceX such intervals have elapsed without the first VM 118 using up its busbandwidth QoS during any given interval, then the first VM 118 isswitched out on the processor 106 for the second VM 118.

It is noted, therefore, that except when the hypervisor 116 has toschedule a VM 118 for running on the processor 106, no software isinvolved while tracking bus bandwidth usage for the currently running VM118 on the processor 106 while this VM 118 is accessing the hardwareresource 110 over the bus 112. Neither the hypervisor 116 nor any of theVMs 118 is involved, in particular. Just hardware is involved intracking such bandwidth usage, particularly the hardware component 108adjusting the bandwidth counter 114, which is implemented directly inhardware and that the component 108 can directly access. When a regulartime interval ends and it is not yet time for a new VM 118 to bescheduled to run on the processor 106, the hardware component 108without involvement of any software resets the bandwidth counter 114 inpart 218.

As such, in at least some implementations, the hypervisor 116 isinvolved only responsive to an interrupt being issued by the hardwarecomponent 108, and more generally when, as noted above, a new VM 118 isto be scheduled for running on the processor 106. From part 210, thehardware component 108 issues an interrupt in part 212 to signal to thehypervisor 116 that the currently running VM 118 has exceeded itsper-regular time interval bandwidth QoS for the hardware resource 110.From part 216, the hardware component 108 may issue an interrupt tosignal to the hypervisor 116 (or the hypervisor 116 may otherwise becomeaware of) that the currently scheduled VM 118 has non-prematurely endedits prescheduled number of regular time intervals on the processor 106(as opposed to premature such ending when the threshold value is reachedin part 210).

Furthermore, as noted above, different VMs 118 can have different busbandwidth QoS, such that different VMs 118 can have differentpredetermined values that are copied to the bandwidth counter 114 inpart 204 and to which the bandwidth counter 114 is reset in part 218.For example, within a computing-as-a-service scenario, customers orclients may not manage their own computing systems but rather pay aservice provider to provide this service as one or more VMs 118 for thesame or different customers. A given customer may pay more for higherbandwidth QoS related to the hardware resource 110 in question, suchthat this customer's VM 118 or VMs 118 are provided with a higherpredetermined value for the bandwidth counter 114.

The various parts of the method 200 are depicted in a particular orderfor illustrative clarity. However, in actuality, at least some parts ofthe method 200 can be more event driven than that which may be inferredfrom the ordering shown in FIG. 2. For example, the event of theprocessor 106 on which the currently scheduled VM 118 is runningaccessing the hardware resource 110 over the bus 112 may trigger part208 to be performed. The event of the bandwidth counter 114 reaching thethreshold value may trigger part 212 to be performed. The event of aregular time interval expiring may trigger part 218 to be performed. Theevent of it being time to schedule a new VM 118 for running on theprocessor 106 may trigger part 212 to be performed.

FIG. 3 shows another example of the computing system 100 that is morespecific than but consistent with the example thereof that FIG. 1depicts. In FIG. 3, the hardware 102, the software 104, the processor106, the hardware component 108, the bus 112, the hypervisor 116, andthe VMs 118 are depicted as in FIG. 1. However, the hardware resource110 is more specifically shown in FIG. 3 as hardware memory 310, whichis one particular type of hardware resource 110. Furthermore, thehardware component 108 is depicted as including two hardware registers320 and 322 that store the bandwidth counter 114 and a predeterminedvalue 324, respectively. Stated another way, the bandwidth counter 114is implemented as the hardware register 320.

It is noted that other implementations of the computing system 100 caninclude some but not all aspects of FIG. 3 in particularizing FIG. 1.For instance, the hardware memory 310 may be replaced by another type ofhardware resource 110, but the hardware registers 320 and 322implemented as depicted in FIG. 3, or vice-versa. Furthermore, thehardware registers 320 and 322 can be a part of hardware other than thehardware component 108, such as the hardware memory 310 itself, theprocessor 106, another hardware resource 110, or other type of hardware.

In FIG. 3, the hardware register 320 thus stores the bandwidth counter114 that has been described in relation to FIGS. 1 and 2 above. Thehardware register 322 stores the predetermined value 324, which is thepredetermined value 324 to which the bandwidth counter 114 is set when anew VM 118 is being scheduled for running on the processor 106, and towhich the counter 114 is reset at the end of a regular time intervalwhen the predetermined threshold has not been reached. The hypervisor116 may set the predetermined value 324 within the hardware register 322when scheduling a new VM 118 for running on the processor 106, which thehardware component 108 may then copy to the bandwidth counter 114 withinthe hardware register 320. By comparison, the hardware component 108resets the bandwidth counter 114 within the hardware register 320 to thepredetermined value 324 within the hardware register 322 at the end of aregular time interval when the predetermined threshold has not beenreached.

FIG. 4 shows the example method 200 by which bus bandwidth of thehardware memory 310 is managed for QoS purposes in a hardware-centricmanner within the computing system 100 of the example of FIG. 3. In FIG.4, parts 202, 206, 212, 214, and 216 are depicted as in FIG. 2. However,part 204 of FIG. 2 has been replaced by part 404 in FIG. 4, and part 418of FIG. 4 may also be considered as being performed within part 204within FIG. 2. Part 208 of FIG. 2 has been replaced by part 408, andpart 210 has been replaced by part 410 in FIG. 4. Furthermore, part 418of FIG. 4 may be considered as being performed in lieu of or for part218 in FIG. 2. Part 418 of FIG. 4 can thus be considered as beingperformed within each of parts 204 and 216 of FIG. 2.

The hypervisor 116 schedules a VM 118 for running on the processor 106(202). The hypervisor 116 further copies the predetermined valuecorresponding to the bus bandwidth QoS of this currently scheduled VM118 for the hardware memory 310 to the hardware register 322, as thepredetermined value 324 (404). Prior to or when a regular time intervalstarts, the hardware component 108, which may be a memory controllerlike an MMU, copies the predetermined value 324 from the hardwareregister 322 to the hardware register 320, as the bandwidth counter 114(418). A regular time interval thus starts (206).

Each time the currently scheduled VM 118, as running on the processor106, accesses the hardware memory 310 over the bus 112, the hardwarecomponent 108 correspondingly decrements the bandwidth counter 114 ofthe hardware register 320 (408). If the bandwidth counter 114 reaches athreshold value of zero as a result of such decrementation (410), thenthe hardware component 108 issues an interrupt (212). This interruptsignals to the hypervisor 116 that the currently scheduled VM 118 is tobe switched out on the processor 106 for a different, new VM 118 (i.e.,a newly scheduled VM 118), such that the method 200 is repeated at part202.

If, however, the bandwidth counter 114 of the hardware register 320 hasnot yet reached zero (410), and if the current regular time interval hasnot yet elapsed or expired (214), then the method 200 proceeds back topart 214. If the bandwidth counter 114 of the hardware register 320 doesnot reach zero during the current regular time interval, then the method200 proceeds from part 410 to part 214, and then to part 216. If it isnot yet time to schedule a new VM 118 (i.e., a newly scheduled VM 118)to run on the processor 106, then the hardware component 108 resets thebandwidth counter 114 of the hardware register 320 to the predeterminedvalue 324 for the currently scheduled VM 118. That is, the method 200proceeds back to part 418, at which the hardware component 108 againcopies the predetermined value 324 from the hardware register 322 to thehardware register 320 to reset the bandwidth counter 114. The method 200then continues with a new regular time interval starting at part 206.

FIG. 4 thus particularizes FIG. 2 in at least the following ways. First,the method 200 as described in relation to FIG. 4 refers to thecomputing system 100 in the example of FIG. 3 as opposed to in theexample of FIG. 1 as is the case with the method 200 as described inrelation to FIG. 2. Second, the method 200 as described in relation toFIG. 4 provides for copying the predetermined value for a VM 118 in part204 of FIG. 2 via copying the predetermined value 324 to the hardwareregister 322 in part 404 and then copying this predetermined value 324from the hardware register 322 to the hardware register 320 as thebandwidth counter 114 in part 418. Similarly, third, the method 200 asdescribed in relation to FIG. 4 provides for resetting the bandwidthcounter 114 in part 218 of FIG. 2 also via copying the predeterminedvalue 324 from the hardware register 322 to the hardware register 320 asthe bandwidth counter 114 in part 418. As with the particularizing ofFIG. 1 via FIG. 3, the particularizing of FIG. 2 via FIG. 4 can includesome but not all aspects of FIG. 4, however.

It is noted that, as can be appreciated by one those of ordinary skillwithin the art, aspects of the present invention may be embodied as asystem, method or computer program product. Accordingly, aspects of theembodiments of the invention may take the form of an entirely hardwareembodiment, an entirely software embodiment (including firmware,resident software, micro-code, etc.) or an embodiment combining softwareand hardware aspects that may all generally be referred to herein as a“circuit,” “module” or “system.” Furthermore, aspects of the presentinvention may take the form of a computer program product embodied inone or more computer readable medium(s) having computer readable programcode embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium include the following: an electrical connection havingone or more wires, a portable computer diskette, a hard disk, a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), an optical fiber, a portablecompact disc read-only memory (CD-ROM), an optical storage device, amagnetic storage device, or any suitable combination of the foregoing.In the context of this document, a computer readable storage medium maybe any tangible medium that can contain, or store a program for use byor in connection with an instruction execution system, apparatus, ordevice.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device. Program codeembodied on a computer readable medium may be transmitted using anyappropriate medium, including but not limited to wireless, wireline,optical fiber cable, RF, etc., or any suitable combination of theforegoing.

In general, a computer program product includes a computer-readablemedium on which one or more computer programs are stored. Execution ofthe computer programs from the computer-readable medium by one or moreprocessors of one or more hardware devices causes a method to beperformed. For instance, the method that is to be performed may be oneor more of the methods that have been described above.

The computer programs themselves include computer program code. Computerprogram code for carrying out operations for aspects of the presentinvention may be written in any combination of one or more programminglanguages, including an object oriented programming language such asJava, Smalltalk, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The program code may execute entirely on the user's computer,partly on the user's computer, as a stand-alone software package, partlyon the user's computer and partly on a remote computer or entirely onthe remote computer or server. In the latter scenario, the remotecomputer may be connected to the user's computer through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, through the Internet using an Internet Service Provider).

Aspects of the present invention have been described above withreference to flowchart illustrations and/or block diagrams of methods,apparatus (systems) and computer program products according toembodiments of the invention. It will be understood that each block ofthe flowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

It is finally noted that, although specific embodiments have beenillustrated and described herein, it will be appreciated by those ofordinary skill in the art that any arrangement calculated to achieve thesame purpose may be substituted for the specific embodiments shown. Thisapplication is thus intended to cover any adaptations or variations ofembodiments of the present invention. As such and therefore, it ismanifestly intended that this invention be limited only by the claimsand equivalents thereof.

We claim:
 1. A method comprising: each time a currently scheduledvirtual machine (VM) accesses a hardware resource over a bus for thehardware resource via the currently scheduled VM running on a processor,adjusting a bandwidth counter associated with usage of the bus for thehardware resource, by a hardware component without involvement of thecurrently scheduled VM or a hypervisor managing the currently scheduledVM, the bandwidth counter to track usage of the bus for the hardwareresource by each VM of a first VM and a second VM on a per-VM basis; andresponsive to the bandwidth counter reaching a threshold value, issuingan interrupt by the hardware component for handling by the hypervisor tomaintain bandwidth quality-of-service (QoS) of bus bandwidth related tothe hardware resource, wherein the hardware resource and the hardwarecomponent are each hardware different than the processor, wherein thebandwidth counter is implemented directly in hardware as opposed tobeing a software-implemented counter, wherein when the first VM and notthe second VM is scheduled to run on a processor, the hardware componentis to reset the bandwidth counter to a first value associated with thefirst VM, and then adjust the bandwidth counter, without involvement ofany VM or the hypervisor, each time the first VM accesses the hardwareresource over the bus when running on the processor, wherein when thesecond VM and not the first VM is scheduled to run on the processor, thehardware component is to reset the bandwidth counter to a second valueassociated with the second VM, and then adjust the bandwidth counter,without involvement of any VM or the hypervisor, each time the second VMaccesses the hardware resource over the bus when running on theprocessor.
 2. The method of claim 1, further comprising: upon expirationof a regular time interval prior to the bandwidth counter reaching thethreshold value, resetting the bandwidth counter to a predeterminedvalue associated with the currently scheduled VM, by the hardwarecomponent without involvement of the currently scheduled VM or thehypervisor, such that no interrupt is issued by the hardware componentand the hypervisor is not involved with resetting the bandwidth counter.3. The method of claim 2, wherein resetting the bandwidth counter to thepredetermined value by the hardware component comprises: copying thepredetermined value from a hardware register directly accessible by thehardware component, by the hardware component without involvement of thecurrently scheduled VM or the hypervisor.
 4. The method of claim 1,further comprising: upon the currently scheduled VM having to beswitched for a newly scheduled VM to now be run on the processor insteadof the currently scheduled VM, resetting the bandwidth counter to apredetermined value associated with the newly scheduled VM, by thehypervisor.
 5. The method of claim 4, further comprising: upon thecurrently scheduled VM having to be switched for the newly scheduled VM,copying the predetermined value to a hardware register implementeddirectly in hardware as opposed to being a software-implementedregister, by the hypervisor, the hardware register directly accessibleby the hardware component.
 6. The method of claim 1, further comprising:prior to the adjusting the bandwidth counter each time the currentlyscheduled VM accesses the hardware resource over the bus for thehardware resource, resetting the bandwidth counter to a predeterminedvalue associated with the currently scheduled VM.
 7. The method of claim6, further comprising: prior to the adjusting the bandwidth counter eachtime the currently scheduled VM accesses the hardware resource over thebus for the hardware resource, copying the predetermined value to ahardware register implemented directly in hardware as opposed to being asoftware-implemented register, by the hypervisor, the hardware registerdirectly accessible by the hardware component.
 8. The method of claim 1,wherein the bandwidth counter is a hardware register directly accessibleby the hardware component.
 9. The method of claim 1, wherein thehardware resource is hardware memory, and the hardware component is ahardware memory controller.
 10. A system comprising: a processor; aplurality of virtual machines (VMs), including a first VM and a secondVM, to run on the processor; a hypervisor to manage the VMs and toschedule the VMs to run on the processor; a hardware resource; a busconnecting the processor to the hardware resource; a bandwidth counterassociated with usage of the bus for the hardware resource to track theusage of the bus for the hardware resource by each VM on a per-VM basisand implemented directly in hardware as opposed to being asoftware-implemented counter; and a hardware component to: when thefirst VM and not the second VM is scheduled to run on the processor,reset the bandwidth counter to a first value associated with the firstVM, and then adjust the bandwidth counter, without involvement of theVMs or the hypervisor, each time the first VM accesses the hardwareresource over the bus when running on the processor; and when the secondVM and not the first VM is scheduled to run on the processor, reset thebandwidth counter to a second value associated with the second VM, andthen adjust the bandwidth counter, without involvement of the VMs or thehypervisor, each time the second VM accesses the hardware resource overthe bus when running on the processor, wherein the hardware resource andthe hardware component are each hardware different than the processor.11. The system of claim 10, wherein the hardware component, responsiveto the bandwidth counter reaching a threshold value, is to issue aninterrupt for handling by the hypervisor to maintain bandwidthquality-of-service (QoS) of bus bandwidth related to the hardwareresource.
 12. The system of claim 10, wherein upon expiration of aregular time interval associated with the first VM prior to thebandwidth counter reaching a threshold value, no interrupt is issued bythe hardware component and the hypervisor is not involved with resettingthe bandwidth counter.
 13. The system of claim 12, further comprisingone or more hardware registers directly accessible by the hardwarecomponent, the hardware register to store the first and the secondvalues, and the hardware component to reset the bandwidth counter bycopying the first and the second values from the hardware register. 14.The system of claim 10, wherein the bandwidth counter is a hardwareregister directly accessible by the hardware component, and wherein thehardware resource is hardware memory, and the hardware component is ahardware memory controller.
 15. The system of claim 10, wherein thefirst VM is scheduled to run on the processor for a first regular timeinterval, and wherein the hardware component is further to: while thefirst VM is running on the processor, and when the bandwidth counterreaches a threshold value, issue an interrupt to terminate running ofthe first VM prematurely and prior to the first regular time intervalexpiring, wherein the second VM is scheduled to run on the processor fora second regular time interval, and wherein the hardware component isfurther to: while the first VM is running on the processor, and when thebandwidth counter reaches a threshold value, issue the interrupt toterminate running of the second VM prematurely and prior to the secondregular time interval expiring.
 16. A non-transitory computer-readabledata storage medium storing code executable by a hardware component toperform a method comprising: when a first virtual machine (VM) and not asecond VM is scheduled to run on a processor, reset a bandwidth counterassociated with usage of a bus connecting the processor to a hardwareresource to a first value associated with the first VM, withoutinvolvement of the first VM or a hypervisor managing the first VM andthe second VM; while the first VM is running on the processor, adjustingthe bandwidth counter each time the first VM accesses the hardwareresource over the bus; when the second VM and not the second VM isscheduled to run on the processor, reset the bandwidth counter to asecond value associated with the second VM, without involvement of thesecond VM or the hypervisor; while the second VM is running on theprocessor, adjusting the bandwidth counter each time the second VMaccesses the hardware resource over the bus, wherein the bandwidthcounter is to track the usage of the bus for the hardware resource byeach VM on a per-VM basis, and is implemented directly in hardware. 17.The non-transitory computer-readable data storage medium of claim 16,wherein upon expiration of a first regular time interval associated withthe first VM prior to the bandwidth counter reaching a threshold value,no interrupt is issued by the hardware component.
 18. The non-transitorycomputer-readable data storage medium of claim 17, wherein uponexpiration of a second regular time interval associated with the secondVM prior to the bandwidth counter reaching the threshold value, nointerrupt is issued by the hardware component.
 19. The non-transitorycomputer-readable data storage medium of claim 18, wherein the first VMis scheduled to run on the processor for a first regular time interval,and wherein the method further comprises: while the first VM is runningon the processor, and when the bandwidth counter reaches a thresholdvalue, issuing an interrupt to terminate running of the first VMprematurely and prior to the first regular time interval expiring. 20.The non-transitory computer-readable data storage medium of claim 19,wherein the second VM is scheduled to run on the processor for a secondregular time interval, and wherein the method further comprises: whilethe second VM is running on the processor, and when the bandwidthcounter reaches the threshold value, issuing the interrupt to terminaterunning of the second VM prematurely and prior to the second regularinterval expiration.